COMMENT ⊗   VALID 00006 PAGES
C REC  PAGE   DESCRIPTION
C00001 00001
C00002 00002	Run 500. times, benchmark in deriv.mcl[tim,lsp] functionn call is (timit)
C00003 00003	∂16-Oct-81  1037	ROD   via MIT-AI 	DERIV  
C00005 00004	∂15-Oct-81  2343	pratt@Diablo (SuNet) 	Dolphin timings   
C00012 00005	∂15-Oct-81  2110	pratt@Diablo (SuNet) 	Dolphin benchmarks
C00013 00006	∂17-Oct-81  1731	pratt@Diablo (SuNet) 	Dolphin timings   
C00018 ENDMK
C⊗;
Run 500. times, benchmark in deriv.mcl[tim,lsp] functionn call is (timit)
Interpreted SAIL MacLisp
(RUNTIME 5.612) 
(GCTIME 0.346) 
T 
(RUNTIME 5.629) 
(GCTIME 0.309) 
T 
(RUNTIME 5.635) 
(GCTIME 0.463) 
T 
Compiled SAIL MacLisp
(RUNTIME 0.591) 
(GCTIME 0.344) 
T 
(RUNTIME 0.588) 
(GCTIME 0.298) 
T 
(RUNTIME 0.587) 
(GCTIME 0.449) 
T 

5000. times, compiled MC MacLisp
(runtime 7.938372)
(gctime 2.58056)
∂16-Oct-81  1037	ROD   via MIT-AI 	DERIV  
To:   RPG at SU-AI, VRP at SU-AI 
I just ran DERIV on  CADR-5. It has 192K of physical memory. I ran  it in
an address space I'd been using all morning. I didn't switch anything off - i.e.
left it listening to the Chaos Net, and running the little "who line" display
at the bottom of the screen. From that display it looked like it was paging madly
as it chewed up cons cells.

Interpreted on 500. callsran it two times - both came in at between 51 and 52 seconds.

Compiled on 5000. calls ran it five times: 50.95, 59.233, 60.766, 59.3, 59.637.

[the first reading seems counter intuitively loer - it had just done the compilation
and loading, and then, I guess switched regions to the default user cons region -
may be some strange behavior of the paging algorithm which happens because the
ratio of consing to computing is so atypically high here.]

Keep in mind that this is an AI lab CADR, not a symbolics A machine, and certainly
not in the class of  a Symbolics 3600. I will try it on a machine with more physical
memory when one is available.

∂16-Oct-81  1052	ROD   via MIT-AI 	P.S.   
To:   RPG at SU-AI, VRP at SU-AI 
They were all wall clock times, not times from internal timers.

∂15-Oct-81  2343	pratt@Diablo (SuNet) 	Dolphin timings   
Date: 15 Oct 1981 23:34:59-PDT
From: pratt at Diablo
To: equip
Subject: Dolphin timings

I have benchmarked two Lisp programs on the Stanford Dolphins.  The result was
an eye-opener for me, but came as no surprise to HPP-er Gordon Novak, who
has ported a large natural language system, ISAAC, from the U. of Texas
2060 Interlisp to Dolphin Interlisp.  Gordon cited a ratio of about 28 
between these two machines for all phases of Interlisp use, namely 
compilation, loading, and execution.  I asked him for an ISAAC demo, so he
proceeded to load his already compiled system, which he told me had taken 50 
minutes to compile on the Dolphin.  We chatted about features of Interlisp to 
while away the 10 minutes it took the Dolphin to load his demo.  He then ran 
ISAAC on a number of physics problems stated in English.  Problem 1 took 45 
seconds.  Gordon said that when running interpreted on the Dolphin this 
problem had taken 180 seconds, as against 6.4 seconds running interpreted on 
the 2060.  He did not have timings for compiled code on the 2060, though 1-2 
seconds is a reasonable guess.  The remaining problems all showed similar 
behavior.

Gordon commented, "This sort of performance really changes your style of using
computers.  You arrange to work on other things while your program is loading,
or to go out for lunch while it is compiling."

My observations were in close agreement with Gordon's, despite my programs
being much smaller and also more CONS-intensive.  One program took derivatives
of algebraic expressions.  Taking the derivative of 3x↑2+ax↑2+ax+5 in Franz
Lisp on Diablo took 3.4 milliseconds, 4.8 milliseconds in a toy Lisp
system on the Sun terminal, and 6.6 milliseconds in UCI-Lisp on Sumex's KI-10.
On the Dolphin it took 160 milliseconds.  (All these timings were obtained by 
running the problem a large number of times, bringing the measured time up to 
the order of many seconds.)

Another program converted logical formulas into disjunctive normal form.  One 
fairly large formula took 21 milliseconds in Franz Lisp, 29 milliseconds on the
Sun, and 1 second on the Dolphin.

Yesterday a contingent from Xerox visited Stanford to discuss Dolphins in
general and Dolphin Interlisp in particular.  One topic was how to port
Interlisp programs to the Dolphin.  Another was a breakdown of Interlisp
performance on the Dolphin; for example CONS was described as taking around 800
milliseconds.  CONS and function-call-and-return were identified as performance
bottlenecks; CAR and CDR however were described as being fast, although no
figures were given for them.  Masinter is optimistic about being able to make
30% or better improvements in the performance of CONS and call-and-return;
meanwhile one should expect particularly bad performance from programs that
make excessive use of CONS and function calls.

One conclusion from all this is that the cost-effective way to get Interlisp
cycles for those users for whom address space is not a problem is to buy a
2060, which gets you the equivalent of nearly 30 Dolphins, not to mention the
advantage of having more than one Dolphin's worth of power on those occasions
when you aren't competing with 30 other Interlisp users.  I readily concede
that users running up against address space limitations have a problem.
However it is far from clear that Dolphins are the solution to that problem for
any but the most patient of users.  If one is to believe Dick Gabriel's
claim that a Vax 11/780 becomes unusable with two Franz Lisp users, one must
also believe that a Dolphin becomes unusable as soon as it has more than 1/25
of an Interlisp user.

The other conclusion is that no one but a dyed-in-the-wool Interlisp enthusiast
is going to make serious use of Interlisp on the Dolphins.  I am a Lisp
enthusiast myself, but not to the extent that I would bother to use a Lisp as
slow as that on the Dolphins.

Symbolics 3600's sound considerably more attractive in the light of these
Dolphin measurements.  I am very anxious to see these measurements taken on the
3600.

			Vaughan

∂15-Oct-81  2110	pratt@Diablo (SuNet) 	Dolphin benchmarks
Date: 15 Oct 1981 21:01:43-PDT
From: pratt at Diablo
To: RPG@SU-AI, VRP@SU-AI
Subject: Dolphin benchmarks

Sorry, haven't put these in a file yet.  DERIV took 800 seconds compiled
against 24 seconds on the Sun for 5000 iterations.  I just timed DNF a few
hours ago and got 3 seconds compiled for three iterations, so 1000 iterations
should be around 1000 seconds as against 29 seconds on the Sun.  Incidentally
compilation yielded a factor of three improvement for DNF.

∂17-Oct-81  1731	pratt@Diablo (SuNet) 	Dolphin timings   
Date: 17 Oct 1981 17:32:02-PDT
From: pratt at Diablo
To: CSD.GENESERETH@SU-SCORE at SUMEX-AIM, CSD.GERRING@SU-SCORE at SUMEX-AIM,
    CSD.HBROWN@SU-SCORE at SUMEX-AIM, CSD.NOVAK@SU-SCORE, CSD.NOVAK@SU-SCORE at SUMEX-AIM,
    RPG@SU-AI, SCHOEN@SUMEX-AIM, equip@SU-HPP-VAX
Subject: Dolphin timings

My apologies for appearing to be bad-mouthing the Dolphin.  I'll try to get my
tests and my remarks both more neutral and more accurate in future.  

Gordon is quite right about not nettling Xerox.  Clearly the goal should be to
constructively encourage Xerox to improve the Dolphin in those areas where 
it hurts the most.  Equally clearly, this need not imply a commitment
on anyone's part to buy Dolphins in significant volume if it is eventually
concluded that they are not sufficiently effective Lisp tools.  Both further 
Interlisp development and further benchmarking (or just plain experience with 
InterlispD), including getting programs adjusted to the Dolphin environment,
are needed before decisions of this sort can be made.

My apologies for getting Gordon's compile-time and load-time ratios wrong, I
misunderstood what he had told me about this.  (The perils of being a
reporter!)  My "800 milliseconds" for CONS was of course a typo for 800
microseconds, .8 seconds for a CONS would be nonsense even on an 8080.  (I 
wonder about that 800 microseconds.  My DERIV program performed 305,000 
conses and took 800 seconds.  If conses take 800 microseconds then that only 
accounts for 250 seconds.  Hence either CONS does NOT dominate the 
computation time of DERIV or CONS takes more like 2.4 milliseconds on the
Stanford Dolphins.)

I agree heartily with Gordon's statement that writing time is more important 
than running time.  That's why I prefer to write in a standard Lisp style,
without spending time worrying about how to transform my programs to eliminate
consing.  If when I use a cons-intensive style my programs slow to a crawl then
I feel justified in lodging a performance complaint.

Gordon's upbeat remarks about using the Dolphin as a Lisp machine (which seemed
much more positive than when I was talking to him on Thursday) would appear
to contradict Dick Gabriel's feeling that a Vax 11/750 would be "unusable" on
performance grounds as a personal Lisp machine.  It would be nice to get some
agreement from the Lisp community as to what level of performance is tolerable
on a personal computer.  (As a Lisp user myself, I lean more to Dick's position
than Gordon's - nice environment notwithstanding, I think I'd find the Dolphin
slower than I could bear.)

	Vaughan